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ABSTRACT 

As the demands for public accountability increase for the higher education, institutions are seeking 
methods for continuous improvement in order to demonstrate quality within programs and processes, 
including those provided through online education. A six round Delphi study was undertaken with 43 
seasoned administrators of online education programs who agreed upon 70 quality indicators that 
administrators should examine within their programs to evaluate quality. A method for scoring was also 
developed. The original set of quality indicators from the Institute for Higher Education Policy study, 
Quality on the Line: Benchmarks for Success in Internet-Based Distance Education (2000) were used as a 
stalling point and determined still valid in 2010, with modifications. The study resulted in a quality 
scorecard for the administration of online education programs. 
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I. INTRODUCTION 

The development of the Internet has forever changed higher education and distance learning programs. 
Prior to its arrival, distance education, also called distance learning or distributed education, used varied 
methods for course delivery such as mail correspondence, telecourses, or satellite delivery, and was 
clearly on the periphery of higher education. When course delivery using the Internet became an 
option — creating the new phrase online education — it wasn’t long before enrollments began to rapidly 
increase and online education became firmly entrenched within higher education. In fact, numerous 
studies cite tremendous growth in online education, which has outpaced that of traditional higher 
education with the majority of accredited institutions now offering distance learning courses [1,2]. 

While some institutions willingly responded to the increased student demand for flexibility and 
convenience, others grudgingly responded because of the increased competition for student enrollment. 
However, after experiencing success with a few online courses, many institutions developed full degree 
programs to be offered completely online. While the online programs were expected to increase student 
access and increase enrollment, both administrators and faculty expressed concern regarding quality [3], 
how should it be measured, and what evaluation methods should be used for continuous improvement 
strategies and accreditation requirements. Today, in light of the public call for accountability, quality 
assurance of educational programs is still one of the greatest challenges in higher education today [4, 5, 
6 ], 

II. BACKGROUND OF THE STUDY 

Like many industries in the 21st century, higher education is finding that demands for accountability [7] 
along with increased competition, stimulate a need for developing quality improvement strategies. In fact, 
a research study by Rice and Taylor [8] found that 88% of the colleges and universities surveyed affirmed 
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they were engaged in some form of continuous improvement strategy and striving toward increased 
quality in all areas of the institution, including distance and online learning programs. The much talked 
about rapid growth of online education programs may be the reason that the regional accreditors began to 
look closely at online programs and their claims of quality. 

Interestingly, many institutions advertise using the word “quality” with online education programs 
because they believe it creates public interest and market advantage. However, quality online education is 
still difficult to define [9] and many have recognized the need for a more comprehensive system for 
evaluation [10]. Unlike industry recognized quality stamps for corporations, such as the Total Quality 
Management criteria for excellence or the Malcolm Baldrige National Quality Award, an instrument is 
yet to exist for online education for measuring quality programs, and facilitating strategic planning and 
program improvement. However, because of the tremendous growth in online education, higher 
education could benefit from an instrument comprised of industry standards endorsed by online education 
administrators. 

Several rubrics do exist for measuring quality online course materials, such as University of Maryland’s 
Quality Matters, California State University-Chico’s rubric for online instruction, and Blackboard’s 
Exemplary Course rubric. In fact, the Quality Matters program is an industry recognized quality seal for 
online course materials and used by many programs in both the United States and other countries. Online 
education administrators could greatly benefit from a quality indicator tool for program administration to 
not only determine program quality but also assist with future goal setting and strategic planning. Online 
education administrators must take the issue of quality seriously because students may go elsewhere in 
search of quality educational programs [11]. 

A research study by the Institute for Higher Education Policy (IHEP) cited a significant need for 
improved research for distance learning programs and quality standards [12]. Commissioned by the 
National Education Association and Blackboard, Inc., the IHEP followed with a second study that 
identified 24 separate quality indicators chosen by various respected online education leaders of higher 
education institutions out of the original 45 indicators provided by a literature search. The latter report, 
Quality on the Line: Benchmarks for Success in Internet-Based Distance Education [13], is still 
abundantly referenced throughout the literature today. 

III. PURPOSE OF THE STUDY 

This study sought to determine if experts in the administration of online education of various types of 
higher education institutions believe the original 24 indicators of quality online education identified by 
the Institute for Higher Education Policy study [13] are still relevant today and if additional indicators are 
needed to identify quality online education programs. The central purpose was the development of a 
scorecard to measure and quantify elements of quality within online education programs in higher 
education that may also support strategic planning and program improvements. The following questions 
guided the research: 

• Are the standards identified in the IHEP study in 2000 still relevant in 2010 for indicating quality 
in online education programs in higher education? 

• What additional standards should be included that address the current industry in 2010? 

• If additional standards are suggested, will they fall into the already identified themes or will new 
themes emerge? 

• What values will be assigned to the recommended standards that will ultimately yield a numeric 
scorecard for measuring quality online education programs from an online education 
administrator’s perspective that could also support strategic planning and program 
improvements? 
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IV. LITERATURE REVIEW SUMMARY 

A review of the literature of quality evaluation of online education programs reveals several 
commonalities among each article or research study [14, 15, 16, 17, 18, 19, 13, 20, 10, 21, 22, 23, 24, 25], 
The institutional commitment, support, and leadership theme and the teaching and learning theme were 
the most used when determining standards for online education programs. The literature focused on the 
quality of teaching and pedagogy far more than the overall quality of programs. Early in the literature, it 
was the overall design of the course that most authors wrote about since courses moved online before 
complete programs. Faculty support was the second most identified theme in quality evaluation. For 
success in teaching online, faculty require support, training, motivation, compensation, and policy. 
Student support and course development were the third most cited themes in the analyzed studies. It is 
interesting that student support was not cited as much as learning effectiveness. Students require the same 
support services that traditional students need; however, it is often more challenging to find ways to 
deliver those services and support in an online environment. 

Technology, organizational/institutional impact, and evaluation were identified in only 6 of the 14 articles 
and studies reviewed. Technology is foundational to the infrastructure of online education and should be 
considered a critical component to quality and success. Cost effectiveness and management and planning 
were only identified three times in the studies and faculty satisfaction, student satisfaction and student 
retention only listed twice out of the 14 examined. A rubric for determining the quality of online program 
administration could not be located within the literature. 

V. RESEARCH DESIGN AND METHODOLOGY 


A. Method 

The Delphi Method, developed by the Rand Corporation in the early 1950s by Norman Dalkey and Olaf 
Helmer [26], was the methodology utilized for this study. While considered suspect by some, many 
researchers have employed the Delphi Method to gain consensus from experts on a given topic because 
“it replaces direct confrontation and debate by a carefully planned, anonymous, orderly program of 
sequential individual interrogations usually conducted by questionnaires” [27], In fact, according to Day 
and Bobeva, “The Delphi is founded upon the use of techniques that aim to develop, from a group of 
informants, an agreed view or shared interpretation of an emerging topic area or subject for which there is 
contradiction or indeed controversy” [28], 

The Delphi Method was selected as the appropriate research method to develop the quality scorecard 
because of its ability “to seek out information which may generate a consensus on the part of the 
respondent group and correlate informed j udgments on a topic spanning a wide range of disciplines” [29], 
Topics or decisions considered to be subjective usually do not have a single correct solution. The 
“affective, emotional, and expressive dimensions of a problem often subordinate the objective, analytical 
quality of a decision” [30]. Because the topic of this study, the quality of online education programs, is so 
subjective, the researcher believes the Delphi process for reiteration improved the overall outcome of the 
quality scorecard and achieved a greater strength of consensus and buy-in from the members of the expert 
panel. 

B. Study Population, Sample Frame and Sampling Plan 

According to Rossman and Eldredge, “A key factor in any Delphi Study is the qualification of the 
population selected to receive the questionnaires” [31]. The study population consisted of online 
education administrators in higher education who were considered experts in the respective field. 
According to Ziglio [32], if the Delphi panel of experts is selected by personal preference of the 
researcher, the overall validity of the study could decrease. Therefore, the sampling frame was identified 
by the Sloan Consortium (Sloan-C), an organization highly respected for its work with quality online 
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education initiatives. For this study, potential panel members were first identified by Sloan-C as 
recognized experts in the administration of online education who met the following criteria: 

• Five or more years of experience as an administrator of online program in higher education 

• Identified by the Sloan Consortium as a respected expert in the field of online education (having 
published or presented) 

• Work at one of the various types of higher education institutions: 

o Community College 
o Public University 
o Private College or University 
o Faith-based College or University 
o For-Profit Institution. 

For this study, 76 experts were invited and 43participated as panel members in the first survey round. 
Table 1 shows the institutional classification for the members of the expert panel. It is important to note 
that more than 83% of the panel members had nine or more years of experience in the administration of 
online education programs. Of the 43 panel members, 56% were from large public institutions. 


Institutional Classification 

Type 

Size 

Total 

Public (4 year) 

Non-profit 

Farge 

24 

Public Community College (2 year) 

Non-profit 

Farge 

2 

Private (4 year) 

Non-profit 

Farge 

4 

Private (4 year) 

For-profit 

Farge 

1 

Private Faith-Based (4 year) 

Non-profit 

Farge 

1 

Public (4 year) 

Non-profit 

Medium 

2 

Private (4 year) 

Non-profit 

Medium 

3 

Private Faith-based (4 year) 

Non-profit 

Medium 

3 

Public (4 year) 

Non-profit 

Small 

1 

Private (4 year) 

Non-profit 

Small 

2 


Table 1. Institutional Classification for Expert Panel Members Who Participated 

C. Instrumentation and Data Analysis 

The majority of Delphi studies use an open-ended questionnaire for collecting data in the initial phase 
[34, 35]; however, since the IHEP quality standards [13] already existed before this study, judgment of 
the 24 quality standards identified by the IHEP study occurred in Delphi Round I. Respondents were also 
invited to suggest additional quality indicators they believed to be relevant for measuring quality in online 
education programs. Therefore, a combination of open-ended and closed questions was used for each 
survey round. 

For this research study, descriptive statistics were formulated and reviewed for each survey using a five- 
point Fikert-scale: 

1 = Definitely Not Relevant 

2 = Not Relevant 
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3 = Slightly Relevant 

4 = Relevant 

5 = Definitely Relevant. 

Mean and median scores along with standard deviation and mode analysis may be used in Delphi studies 
to determine consensus as well as percentage of responses [34, 36, 37]. Many Delphi studies suggest that 
when 60-80% of panelists agree with a survey item, this signifies consensus [38, 39, 40] with a level of 
70% being the most commonly chosen [38]; however, a clear guideline for consensus still did not exist in 
the literature [35]. According to Hsu and Sandford [36], mean and mode analysis are the most favorably 
used in the literature. 

The Delphi Round 1 survey encouraged the expert panel to validate and revise existing IHEP quality 
standards and add new items to indicate quality for inclusion in the Delphi Round II survey. The Delphi 
Round II survey was developed by including all items from the Delphi Round I survey achieving a mean 
score of 4.0 or above and a panel member agreement of 70% or more along with the revision of the 
existing quality standards, and additional quality indicators suggested by the panel of experts. The Delphi 
Round III survey was developed to include items from the Delphi Round II survey that achieved a mean 
score of less than 4.0 but selected by 70% of panel members. The Delphi Round 111 survey included those 
items for further review by the panel of experts. It also invited panel members to suggest further quality 
indicators they felt were missing from the previous round. The Delphi Round IV survey was developed to 
include all items from the Delphi Round III survey that achieved a mean score of less than 4.0 but 
selected by 70% of the panel of experts. The Delphi Round IV survey also requested members of the 
expert panel to suggest possible scoring methods for the quality standards in order create the quality 
scorecard. The Delphi Round V survey was developed to include the scoring methods suggested in the 
Delphi Round IV survey. Those items that did not achieve a mean score 4.0 or better or 70% consensus 
level were fed back to the members of the panel for a re-vote. In Delphi Round V, panel members were 
asked to vote on the best method of scoring, based on their perceptions as administrators for its accuracy 
in evaluating a quality online program. The Delphi Round VI survey was developed to include those 
items from the Delphi Round V survey that were selected by 70% of the panel members as possible 
scoring methods for the quality scorecard but had not yet reached consensus. The Delphi study concluded 
with a fully developed scorecard for quality online education as perceived by online education 
administrators. 

D. Expert Panel Participation 

Seventy-six prospective panel members were identified by the Sloan Consortium as meeting the criteria 
for this research study and were solicited for participation in the study. Forty-three experts in online 
education administration participated in the first survey round. Typical for the Delphi process, 59% of the 
original panel members completed all six rounds of the Delphi survey process. As confirmed by the 
literature, it is difficult to keep a panel of experts fully engaged for 18 weeks. However, the participation 
rate of 86.8% - 97.7% for each round is well above the 70% per round rate that was recommended by 
Hasson, Keeney, and McKenna [34] and Sumsion [42]. 

E. Description of Delphi Rounds 

1. Delphi Round I 

The Delphi Round I results revealed that the members of the expert panel believed that 23 of the 24 IHEP 
quality indicators were still relevant in 2010; however, each indicator received numerous suggestions for 
revisions for the wording of the text. Mean scores ranged from M = 4.00 to M = 4.97. The IHEP quality 
indicator #15 that was not believed to be relevant, “ Students are provided with hands-on training and 
information to aid them in securing material through electronic databases, interlibrary loans, 
government archives, news sendees, and other sources ” had a mean of 3.74, a standard deviation of .912, 
and 66.2% consensus. This did not meet the guidelines for relevance in this study; however, there were 22 
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additional comments and suggested revisions from the panel for this particular quality indicator, and 
seven of those specifically addressed the phrase “hands on” as being questionable. Only the suggested 
revisions were provided in the next survey round since #15 was not determined relevant. The suggested 
revisions for each quality indicator were fed back to the panel in Delphi Round 11 for further analysis with 
an option to keep the original statement without revisions for all but IHEP #15. 

In addition to the 24 IHEP quality indicators being evaluated, the members of the expert panel used two 
open-ended questions in Delphi Round I to provide additional categories of quality indicators and 
individual quality indicators they believed were not included in the original 24 IHEP list of indicators. 
Twenty-nine panel members provided additional comments and suggestions for additional quality 
indicators that were not addressed by the original IHEP 24 standards. The data were examined for content 
analysis and duplicate elements were removed during the data reduction phase. Of the 29 narrative 
responses (most responses contained several suggestions), 80 potential quality indicators were derived 
after all responses were coded and placed into the original IHEP categories until additional categories had 
been approved by the panel. 

Nineteen responses were provided by panel members in response to the request for additional categories 
of quality indicators although not all responses included suggestions for additional categories. From the 
19 responses, 20 additional categories were suggested. Included in these qualitative responses were 
suggestions to change the Institutional Support category to Institutional and Technology Support and also 
a suggestion that these should be two individual categories. This decision was fed back in the next survey 
round. 

2. Delphi Round II 

A total of 38 expert panel members (95.5% response rate) completed the survey in Round II. Delphi 
Round II fed back to the panel of experts the results from Delphi Round I in an attempt to gain consensus 
on all of the IHEP indicator revisions, newly suggested categories, and potential quality indicators. 

The first question addressed the Institutional Support category question from Delphi Round I: Should the 
word Technology be added to the title, making it Institutional and Technology Support, or should the 
category remain titled Institutional Support, or if Technology Support should become a standalone 
category. The majority of responses were split between the following two options: Institutional and 
Technology Support (40% of the panel agreed) or separating them into two categories, Institutional 
Support and Technology Support (40% of the panel agreed) with some written feedback regarding the 
type of technology support was academic or educational. 

Each of the additional 20 categories that were suggested by the panel in Delphi Round I was rated in 
Delphi Round II using the same Likert-scale and a possible additional rating of Not a Category/Theme but 
should be a quality indicator. Only three of the categories received 70% of the panel votes to be returned 
in Delphi Round III: Social and Student Engagement (Mean = 3.81, 70% panel agreement); Accessibility 
(Mean = 4.60, 62.5% panel agreement); and Instructional Design (Mean = 4.03, 60% panel agreement). 
Consensus was not reached in Delphi Round II on the original 24 IHEP indicators or suggested revisions, 
presented in questions #3 - #26. In fact, six additional revisions were suggested to the original IHEP 
indicators through qualitative responses and were added to Delphi Round III survey for five of the 24 
IHEP Indicators. Revisions that did not receive 70% of the panel vote were eliminated and not included in 
Delphi Round III. 

Fourteen of the 80 additional quality indicators suggested by the panel in Delphi Round I were approved 
with a mean of 4.0 or and met the established parameter of having 70% or more of the panel in 
agreement. Of the remaining quality indicators that were previously suggested by the panel, eight were 
eliminated due to receiving low response from the panel (less than 70% of the panel members believed 
they were relevant). The remaining indicators that received 70% of the panel vote were returned for 
another vote in Delphi Round III. 
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3. Delphi Round III 

Thirty-three expert panel members completed the survey in Round 111. In Delphi Round I, the panel 
suggested that the category of Institutional Support should address those standards with the scope of 
support provided by the institution and the Technology Support category should become a standalone 
category. Consensus was achieved by 81.3 for the category to become two distinct categories: 
Institutional Support and Technology Support. 

Three additional categories from Delphi Round II were presented. Two of the three categories received 
consensus in this round: Social and Student Engagement with M = 4.04 and 70.8% consensus and 
Instructional Design with M = 4.27 and 86.7% consensus. Because there was no clear distinction between 
Instructional Design and the already existing Course Development category, the category was renamed to 
Course Development and Instructional Design. The Accessibility category decreased in Mean from 4.60 
in Delphi Round II to 3.86 in Delphi Round III (a quality indicator addressing accessibility in the Student 
Support category was approved in Delphi Round II). 

Fifteen of the original IHEP Indicators were approved with revisions (#1, #2, #6, #9, #10, #12, #13, #14, 
#15, #16, #17, #20, #21, #23, #24). The expert panel determined that the IHEP indicators #18, Technical 
assistance in course development is available to faculty, who are encouraged to use it and #19, Faculty 
members are assisted in the transition from classroom teaching to online instruction and are assessed 
during the process, should be combined into one quality indicator — Technical assistance in course 
development and assistance with the transition to teaching online is provided. 

Also in Delphi Round III, the panel of experts, with 72.7% consensus, determined that the IHEP indicator 
#10, Before starting an online program, students are advised about the program to determine (1) if they 
possess the self-motivation and commitment to learn at a distance and (2) if they have access to the 
minimal technology required by the course design, should be divided into the following two quality 
indicators: Before starting an online program, students are advised about the program to determine if 
they possess the self-motivation and commitment to lecim at a distance and Before starting cm online 
program, students are advised about the program to determine if they have access to the minimal 
technology required by the course design. The panel of experts also determined that the two new 
indicators should be moved from the Course Structure category to the Student Support category. 

Thirteen additional quality indicators suggested by the panel were approved with a mean of 4.0 or and 
met the established parameter of having 70% or more of panel agreement. Seven suggested quality 
indicators were eliminated due to receiving low response from the panel. The remaining indicators that 
received 70% of the panel vote were returned for another vote in Delphi Round IV. 

4. Delphi Round IV 

Delphi Round IV addressed the remaining IHEP indicators (#3, #4, #5, #7, #8, #11, and #22) that the 
panel had yet to reach consensus on, the suggested indicators remaining without consensus, and invited 
the panel to suggest their ideas for potential methods for scoring the quality scorecard. Each of the 
remaining seven indicators achieved consensus with either a revision to the statement or it was left in its 
original form. 

IHEP #4, Guidelines regarding minimum standards are used for course development, design, and delivery, 
while learning outcomes — not the availability of existing technology — determine the technology being used 
to deliver course content, reached consensus with 89.7%. However, the revision suggested by the panel 
was to divide the original indicator into two separate indicators: Guidelines regarding minimum standards 
are used for course developmen t, design, and delivery of online instruction and Technology is used as a 
tool to achieve learning outcomes in delivering course content. The context of the original indicator 
remained the same in context with there being a need for course development guidelines and that learning 
outcomes should drive the course development process, not technology. 
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Of the 31 suggested quality indicators returned to the panel of experts in Delphi Round IV, 17 achieved 
consensus and were moved to the quality scorecard. Fourteen suggested indicators did not reach 
consensus and were retired. 

Delphi Round IV invited the panel of experts to suggest potential methods for scoring the quality 
scorecard. Fifteen of the 30 panel members suggested a total of eight possible methods, identified as 
Methods A, B, C, D, E, F, G, and H (Table 2). The most popular suggestion. Method C, which received 
votes from five panel members, was to allow ten points for each category of quality indicators, thereby 
making the scorecard worth a total of 90 points. 

5. Delphi Round V 

A total of 28 panel members completed the survey in Round V. Consensus was not reached for the 
scoring method, therefore, an additional Delphi round was needed to select a scoring method. Eight 
methods for scoring the quality scorecard were suggested by the panel of experts in Delphi Round IV 
(Methods A, B, C, D, E, F, G, and H). Not one of the scoring methods was agreed upon by 70% of the 
panel. The results of each scoring method, in order of popularity, are: Method C and F received six votes 
of from panel members, which equaled 21.4% of the vote, respectively; Method E received five votes 
from panel members, which was 17.9% of the total vote; and Method A received four votes from panel 
members, which was 14.3% of the total vote. Methods A, C, E, and F received 75% of the total vote from 
panel members and were fed back to the panel of experts to gain consensus in Delphi Round VI. The 
following scoring methods were retired because they did not receive votes from 70% or more of the 
expert panel members: Methods G and H both received 3 votes, which were 10.3% of the panel vote; 
Method B received 1 vote, which was 3.6% of the panel vote; and Method D received 0 votes. 


Suggested Scoring Method 

Frequency of 
Suggestions 
in Round IV 

Percent of 
Panel 
Votes in 
Round V 

Frequency of Votes in 
Round V 

A. One point per quality indicator 

4 

14.3% 

4 

B. Five points per quality indicator 

1 

3.6% 

1 (Retired) 

C. Each category equals a total of 10 
points 

5 

21.4% 

6 

D. Each category equals one point for 
each 

1 

0% 

0 (Retired) 

E. Each indicator equals one point but 
has 3 possible options: Does not 
meet standard (0 points). Partly 
meets standard (.5 point). Meets or 
exceeds standard completely (1 
point). Quality programs must 
achieve 85% of possible points 

1 

17.9% 

5 

F. Each indicator has 3 possible points 

1 

21.4% 

6 


(0 - not observed, 1 - insufficient, 2 
- moderate use, 3 - completely 
meets criteria), then each area must 
have a certain percentage of the 
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points to consider itself worthy of 
meeting the goals of that area 



G. Each Indicator has 3 options: Below 1 

Acceptable Standards (0 points), 

Meets Expected Standards (1 point) 
and Exceeds Standards (2 points) 

10.7% 

3 (Retired) 

H. A simple Likert scale with anchors 1 

to improve reliability 

10.7% 

3 (Retired) 


Table 2. Results of Suggested Scoring Methods of Delphi Round V 


6. Delphi Round VI 

A total of 26 panel members completed the survey in Round VI. Consensus was reached on the method of 
scoring and two of the final six quality indicators were deemed relevant and included in the quality 
scorecard. Consensus was achieved with Method F, Each Indicator has 3 possible points (0 - not 
observed, 1 - insufficient, 2 - moderate use, 3 - completely meets criteria), then each area must have a 
certain percentage of the points to consider itself worthy of meeting the goals of that area, receiving 
73.1% of the total vote (19 of 26 expert panel members selected this method as the best for scoring a 
quality scorecard for online education programs). This round ended the data collection process as a 
quality scorecard for the administration of online education programs was developed with 70 quality 
indicators and a scoring method of up to a possible three points per indicator, with a total score of 210 
points. 

VI. RESULTS 

The following results are organized by the appropriate research question. 

Are the standards identified in the IHEP/NEA study in 2000 still relevant in 2010 for indicating quality in 
online education programs in higher education ? 

The expert panel determined that 23 of the 24 indicators were still relevant today in 2010. Only one of the 
IHEP original standards was not determined relevant; however, the panel agreed upon a revised version of 
the standard to still be included in the quality scorecard. For each original IHEP standard, panel members 
provided revisions to improve relevancy. These suggestions were fed back to the expert panel in 
subsequent rounds to determine whether the original version should still be used as a quality indicator or 
were the suggested revisions more relevant. This resulted in only one of the 24 IHEP standards not being 
revised (IHEP #3), and one more that only had one word change (IHEP #8). The remaining 22 standards 
were slightly-to-moderately revised including two standards being divided into two additional standards. 
IHEP #4 was only slightly changed with the second indicator focusing technology as a tool for achieving 
learning outcomes. IHEP #10 was moved from the Course Structure category to the Student Support 
category but only slightly changed aside from being split into two indicators. 

Table 3 displays the indicators that originated from the IHEP (2000) study and the resulting revision the 
panel determined relevant for today. The most significant revisions were to IHEP #11 and #22. For #11 
(, Students are provided with supplemental course information that outlines course objectives, concepts, 
and ideas, and learning outcomes for each course are summarized in a clearly written, straightforward 
statement), the panel of experts specified that all course information including the syllabus should be 
available to the student at the time of registration. Table 3 also summarizes the differences in each of the 
revised standard from the original IHEP standards. 

Original IHEP Indicator (2000) Revised Indicator (2010) Differences Addressed 
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Institutional Support 


#1. A documented technology 
plan that includes electronic 
security measures (i.e., 
password protection, 
encryption, back-up 
systems) is in place and 
operational to ensure both 
quality standards and the 
integrity and validity of 
information. 

#2. The reliability of the 

technology delivery system 
is as failsafe as possible 


#3. A centralized system 
provides support for 
building and maintaining 
the distance education 
infrastructure. 


1 . A documented technology plan 
that includes electronic 
security measures (e.g., 
password protection, 
encryption, secure online or 
proctored exams, etc.) is in 
place and operational to ensure 
quality standards, adherence to 
FERPA and the integrity and 
validity of information. 

2. The technology delivery 2. Measurable standards are 

systems are highly reliable and in place for technology 
operable with measurable performance 

standards being utilized such as 

system downtime tracking or 
task benchmarking. 


1 . Online exams and 
adherence to FERPA 
guidelines 


3. A centralized system provides 3. Unchanged 
support for building and 
maintaining the distance 
education infrastructure. 

(Unchanged) 


Course Development 


#4. Guidelines regarding 
minimum standards are 
used for course 
development, design, and 
delivery, while learning 
outcomes — not the 
availability of existing 
technology — determine the 
technology being used to 
deliver course content. 


4a. Guidelines regarding minimum 
standards are used for course 
development, design, and 
delivery of online instruction 

4b. Technology is used as a tool to 
achieve learning outcomes in 
delivering course content. 


4a. Split into two statements 


4b. Technology is a tool 


#5. Instructional materials are 
reviewed periodically to 
ensure they meet program 
standards. 


5. Instructional materials, course 
syllabus and learning outcomes 
are reviewed periodically to 
ensure they meet program 
standards. 


5. Course syllabus and 
learning outcomes are 
reviewed 


#6. Courses are designed to 
require students to engage 
themselves in analysis, 
synthesis, and evaluation as 
part of their course and 


6. Courses are designed so that 
students develop the necessary 
knowledge and skills to meet 
learning objectives at the 
course and program level. 


6. Focus is on learning 
outcomes along with 
student engagement 
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program requirements. These may include 

engagement via analysis, 
synthesis and evaluation. 


Teaching And Leaning 


#7. Student interaction with 
faculty and other students is 
an essential characteristic 
and is facilitated through a 
variety of ways, including 
voice-mail and/or e-mail. 


7. Student-to-Student interaction 
and Faculty-to-Student 
interaction are essential 
characteristics and are 
facilitated through a variety of 
ways. 


7. Student to Student and 
Faculty to Student 
interaction was specified 


#8. Feedback to student 

assignments and questions is 
constructive and provided in 
a timely manner. 


8. Feedback on student 

assignments and questions is 
constructive and provided in a 
timely manner, (one word 
change) 


Just one word changed 
“on” 


#9. Students are instructed in the 
proper methods of effective 
research, including 
assessment of the validity of 
resources. 


9. Students learn appropriate 

methods for effective research, 
including assessment of the 
validity of resources and the 
ability to master resources in 
an online environment. 


9. Student learn instead of 
Students are instructed; 
resources in an online 
environment were added 


Course Structure 


#10. Before starting an online 
program, students are 
advised about the program to 
determine (1) if they possess 
the self-motivation and 
commitment to learn at a 
distance and (2) if they have 
access to the minimal 
technology required by the 
course design. 


10a. (Was in Course Structure) 
Divided into two: 

1) Before starting an online 
program, students are advised 
about the program to 
determine if they possess the 
self-motivation and 
commitment to learn at a 
distance. 

10b. Before stalling an online 

program, students are advised 
about the program to 
determine if they have access 
to the minimal technology 
required by the course design. 


10a. Divided into two 
statements. 


10b. Divided into two 
statements 


#11. Students are provided with 
supplemental course 
information that outlines 
course objectives, concepts, 
and ideas, and learning 
outcomes for each course are 


1 1 . The online course site includes 
a syllabus outlining course 
objectives, learning outcomes, 
evaluation methods, textbook 
information, and other related 
course information, making 


1 1 . Specifies syllabus 
available at time of 
registration which 
includes all course 
requirements 
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summarized in a clearly 
written, straightforward 
statement. 

course requirements 
transparent at time of 
registration. 


#12. Students have access to 
sufficient library resources 
that may include a “virtual 
library” accessible through 
the World Wide Web. 

12. The institution ensures that all 
distance education students, 
regardless of where they are 
located, have access to 
library/learning resources 
adequate to support the 
courses they are taking (SACS 
statement). 

12. Adequate support was 
specified 

#13. Faculty and students agree 
upon expectations regarding 
times for student 
assignment completion and 
faculty response. 

13. Expectations for student 

assignment completion, grade 
policy, and faculty response are 
clearly provided in the course 
syllabus. 

13. The word agree was 
removed; expectations 
are provided, not agreed 
upon 

Student Support 

#14. Students receive 
information about 
programs, including 
admission requirements, 
tuition and fees, books and 
supplies, technical and 
proctoring requirements, 
and student support 
services. 

14. Students receive (or have access 
to) information about programs, 
including admission 
requirements, tuition and fees, 
books and supplies, technical 
and proctoring requirements, 
and student support services 
prior to admission and course 
registration. 

14. Access to needed 

information is provided 
prior to admission and 
registration 

#15. Students are provided with 
hands-on training and 
information to aid them in 
securing material through 
electronic databases, 
interlibrary loans, 
government archives, news 
services, and other sources. 

15. Students are provided with 
access to Paining and 
information they will need to 
secure required materials 
through electronic databases, 
interlibrary loans, government 
archives, new services and other 
sources. 

15. Hands On was 
removed; access to 
training was added 

#16. Throughout the duration of 
the course/program, 
students have access to 
technical assistance, 
including detailed 
instructions regarding the 
electronic media used, 
practice sessions prior to the 
beginning of the course, and 
convenient access to 

16. Throughout the duration of the 
course/program, students have 
access to appropriate technical 
assistance and technical support 
staff. 

16. Removed instructions 
for electronic media 
and practice sessions 
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technical support staff. 



#17. Questions directed to 

student service personnel 
are answered accurately and 
quickly, with a structured 
system in place to address 
student complaint. 

17. Student support personnel are 
available to address student 
questions, problems, bug 
reporting, and complaints. 

17. Problems and bug 
reporting was added 

Faculty Support 

#18. Technical assistance in 
course development is 
available to faculty, who are 
encouraged to use it. 

18/19 Combined: Technical 
assistance in course 
development and assistance 
with the transition to teaching 
online is provided [for 
faculty]. 

18. Combined with #19 

#19. Faculty members are assisted 
in the transition from 
classroom teaching to online 
instruction and are assessed 
during the process. 

18/19 Combined: Technical 
assistance in course 
development and assistance 
with the transition to teaching 
online is provided [for 
faculty]. 

19. Combined with #18 

#20. Instructor training and 
assistance, including peer 
mentoring, continues through 
the progression of the online 
course. 

20. Instructors are prepared to 
teach distance education 
courses and the institution 
ensures faculty receive 
training, assistance and 
support at all times during the 
development and delivery of 
courses. 

20. Instructors are prepared 

#21. Faculty members are 
provided with written 
resources to deal with issues 
arising from student use of 
electronically-accessed data. 

21. Faculty receive training and 
materials related to Fair Use, 
plagiarism, and other relevant 
legal and ethical concepts. 

21. Training was added; Fair 
Use, plagiarism, and 
legal and ethical were 
specified 

Evaluation and Assessment 

#22. The program’s educational 
effectiveness and 
teaching/learning process is 
assessed through an 
evaluation process that uses 
several methods and applies 
specific standards. 

22. The program is assessed 

through an evaluation process 
that applies specific 
established standards. 

22. Education effectiveness 
and teaching and 
learning not specified, 
program assessment is 
more general, and it 
should be against 
established standards 

#23. Data on enrollment, costs, 

23. A variety of data (academic 

23. Variety of data 
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and successful/innovative and administrative including academic is 

uses of technology are used to information) are used to frequently used to 

evaluate program regularly and frequently guide changes 

effectiveness. evaluate program effectiveness 

and to guide changes toward 
continual improvement. 


#24. Intended learning outcomes 24. Intended learning outcomes at 24. Program level outcomes 
are reviewed regularly to the course and program level were added 

ensure clarity, utility, and are reviewed regularly to 

appropriateness. ensure clarity, utility, and 


appropriateness. 


Table 3. Final Results of the Original IHEP 24 Indicators 

What additional standards should be included that address the current industry in 2010? 

After the six Delphi survey rounds, the panel of experts suggested a total of 80 potential quality indicators 
and determined that 45 of those suggested indicators were relevant for a scorecard for quality assessment 
of an online education program. Table 4 reports the results for each quality indicator suggested by the 
panel. 


Round II Round 111 Round IV Final 
INSTITUTIONAL SUPPORT CATEGORY Result Result Result Action 


1 . The institution provides documented 
processes and procedures that enable 
distance learning. 


M=3.19 
65% Retired 


Retired 


2. Underlying learning managements systems 


are flexible enough to support emerging 
technologies, e.g. social networking tools, 
mobile devices, Web 2.0, etc. 

M=3.65 

84% 

M=3.35 

Retired 


Retired 

Institutions must provide guidance to faculty 
and students on use of unsupported 
technologies. 

M=3.19 
65% Retired 

- 

- 

Retired 

The institution makes bookstore services 
available to students. 

M=3.39 

72% 

M=3.55 

M=3.62 

Retired 

Retired 

The institution has defined the strategic 
value of distance learning to its enterprise 
and to its relevant parts. 

M=3.59 

76% 

M=3.87 

M=4.03 

Consensus 

Consensus 
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6. The tech plan also needs to consider and 
address vended relationships and, 
especially, support via cloud computing. It 


needs to ensure end to end operability of all 
systems that support distance learning. Also, 
“security measures” are generally handled 
for all campus enterprise systems through an 
LDAP server which authenticates users. 

M=3.05 
62% Retired 



Retired 

7. Policy for Copyright ownerships of course 
materials exists. 

M=4. 16 
95% 

Consensus 

- 

- 

Consensus 

8. The institution has put in place a governance 
structure to enable effective and 
comprehensive decision making related to 
distance learning. 

M=4. 1 1 
92% 

Consensus 

- 

-- 

Consensus 

INSTITUTIONAL SUPPORT CATEGORY 
cont. 

Round II 
Result 

Round III 
Result 

Round IV 
Result 

Final 

Action 

9. Policies are in place to authenticate that 
students enrolled in online courses, and 
receiving college credit are indeed those 
completing the course work 

M=4. 1 1 

95% 

Consensus 

-- 

- 

Consensus 

10. Sustainability and Scalability: A stable 
support mechanism/financial model to 
reduce recreating the same course multiple 
times for example if an instructor leaves the 
university and there is no agreement 
governing the intellectual property that 
would allow the continued use of the course 
materials. 

M=3.66 

82% 

M=3.29 

Retired 

- 

Retired 

1 1 . Students ensured all they need for degree is 
offered in program before enrolling. 

M=3.45 

70% 

M=3.52 

M=3.90 

Retired 

Retired 
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TECHNOLOGY SUPPORT 

Round II 

Round III 

Round IV 

Final 

Result 

Result 

Result 

Action 

12. Appropriate policies are developed, 
reviewed, and disseminated to all 
stakeholders, (moved to Technology Support 
for Round IV) 

M=3.84 

84% 

M=3.9I 

M=3.99 

Retired 

Retired 

13. Faculty, staff, and students are supported in 
the development and use of new 
technologies and skills, (moved to 
Technology Support for Round IV) 

M=3.74 

79% 

M=3.75 

M=4.15 

Consensus 

Consensus 

14. Institution maintains system for backup for 
data availability, (moved to Technology 
Support) 

M=4.03 

90% 

Consensus 

— 

— 

Consensus 

15. The course delivery technology is 
considered a mission critical enterprise 
system and supported as such, (moved to 
Technology Support for Round IV) 

M=3.89 

84% 

M=4.35 

Consensus 

— 

Consensus 

COURSE DEVELOPMENT/ 
INSTRUCTIONAL DESIGN 

Round II 
Result 

Round III 
Result 

Round IV 
Result 

Final 

Action 

16. There is consistency in course development 
for student retention and quality 

M=4. 1 1 
95% 

Consensus 

- 

-- 

Consensus 

17. Instructional design is provided for creation 
of effective pedagogy for synchronous 
sessions. 

M=3.55 

79% 

Retired 

Duplicate 

-- 

Retired 

18. Curriculum development is a core 
responsibility for faculty. 

M=3.32 

74% 

M=3.45 

M=4.03 

Consensus 

Consensus 

19. Learning objectives describe outcomes that 
are measurable. 

M=3.82 

79% 

M=4.32 

Consensus 

-- 

Consensus 

20. Development of online course materials 
takes into account the changing context of 
media delivery 

M=3.55 

84% 

M=3.75 

M=3.93 

Retired 

Retired 
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2 1 . Selected assessments measure the course 
learning objectives and are appropriate 
for an online learning environment 

M=3.92 

84% 

M=4.32 

Consensus 

— 

Consensus 

22. Course objectives provide opportunity 
for student interaction. 

M=3.84 

78% 

M=3.77 

Retired 

— 

Retired 

23. Course design promotes both faculty and 
student engagement. 

M=4. 1 6 

86% 

Consensus 

— 

— 

Consensus 

24. Student-centered instruction is 
considered during the course- 
development process. 

M=4.03 

92% 

Consensus 

— 

— 

Consensus 

25. Instructional design is provided for 
creation of effective pedagogy for both 
synchronous and asynchronous class 
sessions. 

M=3.84 

84% 

M=3.84 

M=4.24 

Consensus 

Consensus 

26. Current and emerging technologies are 
evaluated and recommended for online 
teaching and learning. 

M=3.87 

92% 

M=3.91 

M=4.10 

Consensus 

Consensus 

TEACHING AND LEARNING 

Round II 
Result 

Round III 
Result 

Round IV 
Result 

Final 

Action 

27. Students are provided access to library 
professionals and resources that help 
them to deal with the overwhelming 
amount of online resources. 

M=3.39 

79% 

M=3.58 

M=4.00 

Consensus 

Consensus 

28. Course material presented in a variety of 
ways 

M=3.42 

82% 

M=3.52 

M=3.82 

Retired 

29. Interactive elements such as video and 
flash graphics to help engage the 
students’ understanding of key learning 
objectives 

M=3.30 

76% 

M=3.42 

M=3.46 

Retired 
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30. Students are provided access to library 
professionals and resources that help 
them to deal with the overwhelming 
amount of online resources. 

M=3.1 1 

69% 

Duplicate 

Retired 

— 

Retired 

3 1 . Online courses/programs use one course 
management platform, creating a single 
delivery model, and students receive an 
online instructional orientation to the 
course management platform. 

M=3.66 

79% 

M=3.81 

M=3.86 

Retired 

COURSE STRUCTURE 

Round 11 
Result 

Round III 
Result 

Round IV 
Result 

Final 

Action 

32. Instructors use specific strategies to 
create a presence in the course. 

— 

Presented 
Round VI 

M=4. 12 
Consensus 
Round VI 

Consensus 

33. Opportunities/tools provided to 

encourage student-student collaboration 
(i.e, web conferencing, instant 
messaging, etc). 

M=3.50 

76% 

M=3.81 

M=4. 14 
Consensus 

Consensus 

34. Honor code used to enable a culture of 
accountability 

M=3.39 

76% 

M=3.19 

Retired 

— 

Retired 

35. Links or explanations of technical 
support are available in the course. 

M=3.95 

87% 

M=4.29 

Consensus 

— 

Consensus 

36. Instructional materials are easily 
accessible and usable for the student. 

M=4.26 

89% 

Consensus 

— 

— 

Consensus 

37. The course adequately addresses the 
special needs of disabled students via 
alternative instructional strategies and/or 
referral to special institutional resources. 

M=4.29 

95% 

Consensus 

— 

— 

Consensus 

38. Optional synchronous sessions with 
faculty are offered and archived to be 
available asynchronously as well, to 
allow students access to faculty 

M=3.1 1 

68% 

Retired 


Retired 
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39 . Documents attached to modules are in a 
format that is easily accessed with 
multiple operating systems and 
productivity software (PDF. for 
example). 

— 

Presented 
Round VI 

M=4.32 
Consensus 
Round VI 

Consensus 

40. Each course includes an orientation 
module. 

— 

Presented 
Round VI 

M=3.64 
Retired 
Round VI 

Retired 

41. Students have at least some choice in 
their activities/assignments. 

— 

Presented 
Round VI 

M=2.92 
Retired 
Round VI 

Retired 

42 . Course modules are designed for visual 
appeal as well as clarity and consistency 
(use of white space, color, well-chosen 
fonts, no gimmicky graphics/animations 
that have no real puipose. 

— 

Presented 
Round VI 

M=3.60 
Retired 
Round VI 

Retired 

43 . Institution branding is evident in every 
part of each course. 

— 

Presented 
Round VI 

M=3.08 
Retired 
Round VI 

Retired 

STUDENT SUPPORT 

Round 11 
Result 

Round III 
Result 

Round IV 
Result 

Final 

Action 

44. Students are provided relevant 

information: ISBN numbers, suppliers, 
etc. and delivery modes for all required 
instructional materials: digital format, e- 
packs, print format, etc. to ensure easy 
access. 

M=3.50 

76% 

M=3.94 

M=4. 14 
Consensus 

Consensus 

45. While technologies may not be supported 
centrally (like available in the cloud or 
openly), there needs to guidance on how 
these tools will be supported and the 
ramifications to students. 

M=3.05 

71% 

M=3.35 

M=3.31 

Retired 

Retired 
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46. Student support services are provided for 
outside the classroom such as academic 
advising, financial assistance, peer 
support, etc. 

M=4.05 

89% 

Consensus 

— 

— 

Consensus 

47. Program demonstrates a student-centered 
focus rather than trying to fit service to 
the distance education student in on- 
campus student services. 

M=3.79 

79% 

M=3.81 

M=4.07 

Consensus 

Consensus 

48. Automated support tools are available for 
faculty to provide early intervention to 
support student success. 

M=3.51 

81% 

M=3.55 

M=3.69 

Retired 

49. Efforts are made to engage students with 
the program & institution 

M=3.58 

79% 

M=3.84 

M=4.07 

Consensus 

Consensus 

50. Students are instructed in the appropriate 
ways of communicating with faculty and 
students 

M=3.68 

82% 

M=3.87 

M=4.21 

Consensus 

Consensus 

5 1 . Students are instructed in the appropriate 
ways of enlisting help from the program. 

M=3.50 

74% 

M=3.71 

M=4.33 

Consensus 

Consensus 

52. Support services designed to build 

communication and affiliation among the 
online student population 

— 

— 

M=3.63 

Retired 

Retired 

53. Students agree and understand the 

expectations of the program and courses 

M=3.66 

79% 

M=3.90 

M=3.97 

Retired 

Retired 

54. Students should be provided a way to 
interact with other students in an online 
community 

M=3.42 

74% 

Duplicate 

Retired 

— 

— 

Retired 

55. The institution provides guidance to both 
students and faculty in the use of all 
forms of technologies used for course 
delivery 

M=3.44 

71% 

M=3.77 

M=4.21 

Consensus 

Consensus 
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56. Students have access to effective 
academic, personal, and career 
counseling 

M=3.82 

87% 

M=4.19 

Consensus 

- 

Consensus 

57. Tutoring is available as a learning 
resource. 

M=3.89 

92% 

M=3.94 

M=4.07 

Consensus 

Consensus 

58. Minimum technology standards are 
established and made available to 
students. 

M=3.97 

82% 

M=4.13 

Consensus 

-- 

Consensus 

59. Policy and process is in place to support 
ADA requirements. 

M=4. 16 

87% 

Consensus 

- 

- 

Consensus 

SOCIAL AND STUDENT ENGAGEMENT 

Round II 
Result 

Round III 
Result 

Round IV 
Result 

Final 

Action 

60. Students should be provided a way to 
interact with other students in an online 
community. 

M=3.61 

79% 

M=3.94 

M=4.07 

Consensus 

Consensus 

FACULTY SUPPORT 

Round II 
Result 

Round III 
Result 

Round IV 
Result 

Final 

Action 

6 1 . New learning skills for online teaching 
and learning are identified. 

M=3.30 

76% 

M=3.50 

M=3.62 

Retired 

Retired 

62. Review of web. 2.0 tools and emerging 
technologies and faculty. 

M=3.14 

73% 

M=3.35 

M=3.31 

Retired 

Retired 

63. Workshops are provided for keeping 
faculty updated in selection and use of 
tools. 

64. Faculty are provided ongoing 
professional development related to 
online teaching and learning. 

M=3.57 

81% 

Duplicate 

Retired 

M=4. 16 

87% 

Consensus 

- 

- 

Retired 

Consensus 

65. Faculty workshops are provided to make 
them aware of emerging technologies 
and the selection and use of these tools. 

M=3.50 

76% 

M=3.77 

M=4.03 

Consensus 

Consensus 

66. Clear standards are established for 
faculty engagement and expectations 
around online teaching 

M=4.05 

84% 

Consensus 

- 

- 

Consensus 
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EVALUATION AND ASSESSMENT 

Round II 
Result 

Round III 
Result 

Round IV 
Result 

Final 

Action 

67. Online learning should be robustly 

evaluated using tools widely available, so 
that faculty and students know what 
students perceive about the efficacy of 
online learning and so the institution 
knows how they compare and how they 
can improve. 

M=3.42 

71% 

M=3.55 

M=3.71 

Retired 

Retired 

68. A process is in place for the assessment 
of faculty and student support services. 

M=3.97 

87% 

M=4.26 

Consensus 

— 

Consensus 

69. Course and program retention is 

assessed. Results of course evaluations 
are used as part of faculty/instructor 
performance evaluations. 

M=3.84 

84% 

M=4. 19 
Consensus 

— 

Consensus 

70. Recruitment and retention are examined 
and reviewed 

M=3.55 

76% 

M=4.06 

Consensus 

— 

Consensus 

7 1 . Evaluation should include evaluation by 
potential employers. 

M=2.76 

55% 

Retired 

— 

— 

Retired 

72. Course evaluations collect student 
feedback on quality of content and 
effectiveness of instruction. 

M=4.03 

89% 

Consensus 

— 

— 

Consensus 

73. The relationship between online 
education programs and institutional 
mission must be included as a measure. 

M=3.32 

71% 

M=3.48 

M=3.41 

Retired 

Retired 

74. Program demonstrates compliance and 
review of accessibility standards (Section 
508, etc.). 

M=3.82 

84% 

M=4.29 

Consensus 

— 

Consensus 

75. Student evaluations of 
course/instructor/program are made 
available. 

76. Course evaluations are examined in 
relation to faculty performance 
evaluations. 

M=3.43 

70% 

M=3.68 

82% 

M=3.86 

M=4.00 

Consensus 

M=3.86 

Retired 

Retired 

Consensus 

77. Aggregation of data to ensure each class 
is being taught well. 

M=3.21 

66% 

Retired 

-- 

— 

Retired 
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78. 

Faculty performance is regularly 

M=3.84 

M=4.39 


assessed. 

79% 

Consensus 

79. 

Alignment of learning outcomes from 
course to course exists. 

M=3.63 

M=4.26 



79% 

Consensus 


80. Online learning should be robustly 

evaluated using tools widely available, so 

that faculty and students know what 

students perceive about the efficacy of 

online learning and so the institution 

knows how they compare and how they M=2.84 

can improve. The credentials of the 57% 

distance education support staff and Retired 

administration, in terms of years of 

professional experience and education 

level as well as type of degree earned 

(educational technology or general 

education verses non-education). 


Consensus 


Consensus 


Retired 


Table 4. Suggested Quality Indicators 

If additional standards are suggested, will they fall into the already identified themes or will new themes 
emerge ? 

The majority of the additional standards suggested by the experts did indeed fall naturally into the 
existing seven IHEP Categories: Institutional Support, Teaching and Learning, Student Support, Faculty 
Support, Course Structure, Course Development, and Evaluation and Assessment. It is important to point 
out that in the original IHEP list of quality indicators, the Institutional Support category primarily 
addressed technology support standards and not necessarily those related to institutional support such as 
mission and strategic planning; therefore, the panel of experts determined two categories were necessary: 
Technology Support and Institutional Support. The existing IHEP indicators in the Institutional Support 
category were moved to the Technology Support since their focus was technology support provided by 
the institution. 

Aside from dividing the Institutional Support and Technology Support categories, the panel of experts 
suggested an additional 20 categories but only 2 of those suggestions achieved consensus: Instructional 
Design and Social and Student Engagement. The researcher combined Instructional Design with the 
Course Development category, now called Course Development and Instructional Design, because there 
lacked clear distinction for identifying quality indicators for either category. After all panel voting had 
concluded, the Technology Support and Social and Student Engagement category were the only two new 
categories added to the Scorecard; however, it is interesting to note there was only one quality indicator in 
Social and Student Engagement category that achieved panel consensus. 

At the conclusion of the study, nine categories of quality indicators existed: Institutional Support, 
Technology Support, Faculty Support, Course Structure, Course Development and Instructional Design, 
Teaching and Learning, Student Support, Social and Student Engagement, and Evaluation and 
Assessment. 

What values will be assigned to the recommended standards that will ultimately yield a numeric 
scorecard for measuring quality online education programs from an online education administrator’ s 
perspective that could also support strategic planning and program improvements? 

Eight potential scoring methods were suggested in Delphi Round IV. After voting in Delphi Round V 
concluded, four of the methods were removed for lack of consensus. Only those selected by 70% of the 
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panel were reviewed again by the panel of experts. The panel of experts determined that each quality 
indicator should be worth a potential three points for a total of 210 points. Each quality indicator will be 
scored in the following manner: 0 points - not observed, 1 point - insufficient, 2 points - moderate use, 3 
points - completely meets criteria. The panel had also suggested that a parameter or a minimum score be 
established for each category of the scorecard (a certain percentage of the points) to establish a goal; 
however, the panel did not make a suggestion as to what the minimum score for each category should be. 


VII. IMPLEMENTATION AND NEXT STEPS 

The quality scorecard is versatile enough to be used to demonstrate the overall quality of online education 
programs, no matter what size or type of institution. The following steps for use and implementation are 
suggested that will yield a measurable result: 

1 . The online education administrator examines the online program for evidence of each of the 70 
quality indicators. Based upon the level of evidence observed, the administrator chooses one of 
the following values: 0 points - not observed, 1 point - insufficient, st2 points - moderate use, 3 
points - completely meets criteria. 

2. For each indicator, the online education administrator should provide examples of the observed 
evidence. For example, the first indicator listed in the Institutional Support category is: The 
institution has put in place a governance structure to enable effective and comprehensive 
decision making related to distance learning. To substantiate the score for this indicator, evidence 
should be documents such as digital copies of organizational charts, reporting structures, and 
advisory committee minutes demonstrating how a decision is processed. 

3. The online education administrator totals the score for each indicator and then determines the 
level of quality observed: 

A perfect score = 210 points. 

90-99% = 189-209 - Exemplary (little improvement is needed) 

80-89% = 168-188 - Acceptable (some improvement is recommended) 

70-79% = 147-167 - Marginal (significant improvement is needed in multiple areas) 

60-69% = 126-146 - Inadequate (many areas of improvement are needed throughout the program) 
59% and below = 125 points and below - Unacceptable 
The quality scorecard tool resulting from this research study is available on the Sloan Consortium (Sloan- 
C) website at http://sloanconsortium.org/quality_scoreboard_online_program. It is the intent of the author 
and Sloan-C to make the scorecard interactive so that administrators of online education programs may 
use the scorecard tool to demonstrate program evaluation. An ancillary handbook for use and 
implementation of the quality scorecard is being developed to better guide the administrator in its use. 
Each of the seventy quality indicators will be defined with more depth and examples and best practices 
will be provided to better demonstrate the level of quality that may be reached with each indicator. A 
community of practice website has been developed by Sloan-C that provides a forum for notes and 
queries to be shared regarding the scorecard and process for program evaluation. 


VIII. CONCLUSION 

The purpose for this study was the development of a scorecard to measure and quantify elements of 
quality within online education programs in higher education. Quality is a perception that varies within 
industries, including that of higher education whose traditional indicators for quality are changing. In fact, 
Pond observed, 

It is quite clear that education in the 21st century presents challenges to quality assurance 
that were unimaginable just a quarter century ago. E-learning in particular, with its ability 
to render time and place irrelevant, requires that we abandon traditional indicators of 
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“quality” such as “contact hours,” “library holdings,” and “physical attendance” among 
others in favor of more meaningful measures. [43] 

As we abandon the traditional indicators we have used for so long, higher education needs a method to 
identify and assess quality within online education programs that could provide a method of 
benchmarking and a path to improvement. This study provides just such a process by creating a scorecard 
for the administration of quality online education programs. The study also extends further validity to the 
original 24 IHEP indicators [13], in spite of it being a decade later. The original IHEP research study 
identified a strong base of quality indicators that, for the most part, have withstood the test of many 
changes throughout the field of online education. The original indicators are all included in the quality 
scorecard, although, all but two were revised without the primary focus being changed. 

While there are rubrics being used to assess quality in online course materials, until now, there was not an 
industry agreed upon instrument being used to evaluate online education programs. Many institutions 
prolifically advertise they offer quality online education but have not had a way to quantify or benchmark 
their programs. How do students know they are enrolling in a quality program? The scorecard developed 
as a result of this research study provides an instrument that could identify strengths and weaknesses of an 
online education program and be used as a benchmarking tool for evaluation against other like programs 
in the industry. 

The identification of quality online education programs satisfies a great need in our field and has been 
requested by many online education administrators as a tool for program improvement. The assessment of 
quality online education has never been more important as fierce competition from for-profit programs as 
well as many non-profits programs continues to increase and students all over the world are clicking to 
find a respectable degree program. Quality online education really does matter as the ultimate impact is to 
our students. 
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